Compression and decompression of reference images in a video encoder

ABSTRACT

Methods and devices transform image data, which are transformed by a compression filter before being compressed and stored in a reference image memory. In an extension, an inverse transformation to that of the compression filter is performed by a decompression filter when image data from the reference memory are read out and decompressed. The methods and devices can be used for image compression methods and image decompression methods that use reference image memories.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based on and hereby claims priority to InternationalApplication No. PCT/EP2012/050435 filed on Jan. 12, 2012 and EuropeanApplication No. 11150714.1 filed on Jan. 12, 2011, the contents of whichare hereby incorporated by reference.

BACKGROUND

The invention relates to methods and devices for transforming firstframe data.

Over the years, frame formats to be encoded have become increasinglylarge due, for example, to the introduction of new types of recordingsystems, including the current changeover from a PAL (Phase AlternatingLine) TV transmission system used in Europe over the last 50 years,having a 625×576 pixel frame size, to a 1920×1080 or 1280×720 pixel HDTV(High Definition Television) resolution. In the future, even largerframe formats are expected to be introduced in new types of TV systems.

HDTV and future systems use digital compression methods to compress asequence of video frames so that they can be transmitted over theInternet or mobile communications channels for example. However, due tothe increased frame format sizes, the computing power required forcompressing the sequence of video data and the amount of memory requiredhere are increasing significantly. One consequence of this is that datatransfer between memory and processing units which implement thecompression methods is also increasing significantly.

Therefore study groups such as the Joint Collaborative Team on VideoCoding (JCT-VC), a common working party of the ITU and ISO/IEC(ITU—International Telecommunication Union, ISO—InternationalStandardization Organisation, IEC—International ElectrotechnicalCommission) are working not only on improving the compression rate, butalso on standardized methods for enabling video frames to be efficientlystored in reference frame buffers of the respective codec and accessedin a resource-saving manner.

FIG. 1 shows a known device for compressing a sequence of frames,comprising a reference frame buffer SRB. Frames are encoded, forexample, using predictive coding, also know as inter-frame coding. Oneof the frames is divided into frame blocks BB, e.g. of 16×16 pixels, andis then encoded frame block by frame block. Then for one of the frameblocks a reference frame block RBB providing a good basis for estimatingthe content of the frame block is searched for in a reference frame REF.For this purpose the frame block is transferred to a motion estimationunit ME which, on the basis of a reference sub-frame REFT comprisingparts of the reference frame REF after frame decompression by a framedecompression unit PD, selects the reference frame block from thereference sub-frame and signals the selected reference frame block to amotion compensation unit MC by a motion vector MV. The motioncompensation unit provides the reference frame block on the basis of thereference frame and motion vector.

In a next step, a difference frame block BD is generated by subtractingthe reference frame block RBB from the frame block BB. The differenceframe block subsequently undergoes transformation in a transformationunit T, e.g. using a discrete cosine transform method. Available at theoutput of the transformation unit are transform coefficients TK whichare then fed to a quantization unit Q for quantization. Available at theoutput of the quantization unit are quantized transform coefficients TQwhich are converted into an output signal AS by entropy encodingperformed by an entropy encoding unit EC.

In a feedback loop, the quantized transform coefficients TQ areconverted into reconstructed transform coefficients TKR by inversequantization by an inverse quantization unit IQ. These reconstructedtransform coefficients TKR are transformed into a reconstructeddifference frame block BDR by inverse transformation by an inversetransformation unit IT. In a further step, a reconstructed frame blockRBM is generated by adding the reconstructed difference frame block BDRand the reference frame block RBB.

In older encoding methods, the reconstructed frame block is writtendirectly to the reference frame buffer. In methods currently instandardization, to reduce a data volume the reconstructed frame blockfirst undergoes frame compression by a frame compression unit PC whichsignificantly reduces the data volume of the reconstructed frame block.A compressed reconstructed frame block RBC produced by the framecompression unit PC is then stored in the reference frame buffer. Inorder to enable the motion estimation unit and the motion compensationunit to access the required frame data, when a reference frame REF orrather a specific section of the reference frame is requested, therespective compressed reconstructed frame block is read out from thereference frame buffer SRB and converted into a reference sub-frame REFTby frame decompression by a frame decompression unit PD.

FIG. 2 shows a decoder corresponding to the encoder shown in FIG. 1. Theoutput signal AS is decoded into quantized transform coefficients TQ byan entropy decoding unit ED. In addition, the quantized transformcoefficients are inversely quantized into reconstructed transformcoefficients TKR by the inverse transformation unit IQ. This is followedby inverse transformation of the reconstructed transform coefficientsTKR into a reconstructed difference frame block BDR by the inverseinformation unit IT. In addition to the output signal, the respectivemotion vector MV, among other things, is also transmitted to thedecoder. Using the reference sub-frame REFT, the decoder can determinetherefrom, by the motion compensation unit MC, the reference frame blockRBB which is converted into the reconstructed frame block RBM by addingit to the reconstructed difference frame block.

The reconstructed frame block RBM can be reproduced on a display, forexample. The reconstructed frame block RBM is then converted bycompression by the frame compression unit PC into the compressedreconstructed frame block RBC which is then stored in the referenceframe buffer SRB. The compressed reconstructed frame blocks stored inthe reference frame buffer can be decompressed into the referencesub-frame by the frame decompression unit PD.

The article by Chong Soon Lim (“Reference Frame Compression using ImageCoder,” 2010) describes a lossless frame compression/decompressionmethod in which bit-plane coding is carried out after floating pointdiscrete cosine transformation (DCT) and scanning of coefficients in aone-dimensional representation, arranged two-dimensionally after thetransformation.

In a method according to Mehmet Umut Demircin et al. (“CompressedReference Frame Buffers [CRFB],” 2010), a buffer access bandwidthreduction technique is proposed. In addition to transformation andquantization, DC prediction and entropy encoding for the framecompression unit PC and/or a reverse step for the frame decompressionunit PD are also proposed.

Madhukar Budagavi (ALF Memory Compression and IBDI/ALF coding efficiencytest results in TMuC-0.1,” 2010) describes test results for compressionand decompression of frame data upstream and downstream respectively ofa deblocking frame memory.

Lastly, Hirofumi Aoki (“DPCM-Based Memory Compression,” 2010) describesa one-dimensional DPCM-based frame memory compression method(DPCM—discrete pulse code modulation).

At least the compression methods proposed by Lim and Aoki are lossless.

SUMMARY

One potential object is to specify methods and devices with whichcompression and decompression by the frame compression unit and framedecompression unit respectively can be increased compared to the relatedart.

The inventors propose a method for transforming first frame data which,having been compressed by a frame compression unit, is stored in areference frame buffer and, on retrieval from the reference framebuffer, is decompressed into second frame data by a frame decompressionunit, wherein prior to compression the frame data is transformed by acompression filter such that an increase in the frame compression unit'scompression rate is produced.

In existing coding methods the frame compression unit often operateslosslessly. By using the compression filter, loss of image detail can becontrolled by a filter parameter of the compression filter withoutinteraction with the frame compression unit. It is also advantageousthat the compression filter design can be individually adapted to thecharacteristics of the frame compression unit.

The compression filter is preferably generated as a function of at leastone of the following parameters:

-   -   a coding mode used for encoding the first frame data;    -   a quantization parameter used for encoding the first frame data;    -   a motion vector used for predictive coding of the first frame        data

Controlling the compression filter as a function of at least one of theabove mentioned parameters enables improved compression to be achieved,as it enables specific characteristics of the first frame data to becontrolled, such as a specific coding mode, a quantization parameterused or a motion vector employed.

In an advantageous development, the compression filter takes the form ofa subband filter, in particular with suppression of quantization noisepresent in the first frame data as a result of quantization. Thisenables compression by the frame compression unit to be improved in thatimage content which can only be encoded with great complexity, such assharp-edged lines, and “useless” image content such as quantizationnoise, are filtered and can therefore be compressed more efficiently.

The compression filter is advantageously determined from a range of aplurality of options such that the filter which optimizes an errorfunction for a specifiable data rate of the first frame data, i.e.minimizing the error by compression using the compression error at afixed data rate, is selected. Only the compression filter which achievesa very good image quality at a specifiable data rate is therebyselected. Similarly, this development can also be implemented such thatthe filter producing the lowest data rate without quality impairment isselected.

The second frame data is preferably generated such that frame datadecoded after decompression by the decompression unit is subject to adecompression filter, wherein reverse steps to the compression filterare carried out by the decompression filter.

This development enables compression filters to be used which onlyreduce the sharpness of the first frame data. For this purpose, thesecond frame data is generated such that frames decoded afterdecompression by the frame decompression unit are subject to adecompression filter, wherein the compression filter and thedecompression filter perform at least one of the following operations:

-   -   a first number of pixels of the first frame data is reduced by        the compression filter and a second number of pixels of the        second frame data is increased by the decompression filter such        that the first number and the second number assume an identical        value;    -   a third number of bit planes of the pixels of the first frame        data is reduced by the compression filter and a fourth number of        bit planes of the pixels of the second frame data is increased        by the decompression filter such that the third number and the        fourth number assume an identical value.

This enhancement enables the first frame data to be transformedparticularly efficiently, so that compression can be performed extremelyefficiently by the frame compression unit.

The inventors also propose devices for transforming first frame datawhich, having been compressed by a frame compression unit, is stored ina reference frame buffer and is decompressed into second frame data by aframe decompression unit on retrieval from the reference frame buffer,wherein the device has a compression filter for transforming the framedata, the filter transforming the first frame data prior to compressionso as to produce a compression rate increase in the frame compressionunit. Advantages of these devices correspond with those of the method.

The device is preferably supplemented by a decompression filter forgenerating the second frame data such that frame data decoded afterdecompression has been performed by the frame decompression unit issubject to the decompression filter, the decompression filter operatinginversely to the compression filter. Advantages of this enhancement areset forth in this document with regard to the corresponding method.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects and advantages of the present invention willbecome more apparent and more readily appreciated from the followingdescription of the preferred embodiments, taken in conjunction with theaccompanying drawings of which:

FIG. 1 shows a related art encoder for compressing frame data;

FIG. 2 shows a related art decoder for decompressing compressed framedata;

FIG. 3 shows a first modified encoder representing a first variant ofthe inventors' proposals;

FIG. 4 shows a first modified decoder representing the first variant ofthe inventors' proposals;

FIG. 5 shows a second encoder representing a second variant of theinventors' proposals;

FIG. 6 shows a second decoder representing the second variant of theinventors' proposals.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to like elementsthroughout.

Elements having an identical function and mode of operation are providedwith the same reference characters in the figures.

FIGS. 1 and 2 show a frame compression unit PC and frame decompressionunit PD used according to the related art. As FIGS. 1 and 2 have beenexplained in detail in the introduction, reference is made here to theremarks in the introduction.

FIG. 3 shows a first exemplary embodiment of a transforming method, theFIG. 3 likewise representing a device DEV for carrying out this method.FIG. 3 differs from FIG. 1 in that a compression filter CF isincorporated in the signal processing chain immediately upstream of theframe compression unit PC. The reconstructed frame block RBM, alsotermed the first frame data, is fed to the compression filter CF fortransformation, i.e. filtering. After transformation has been performed,the first frame data RBM is present at the output of the compressionfilter CF as modified first frame data RBMX. This modified first framedata is then fed to the frame compression unit PC to compress themodified first frame data into the compressed reconstructed frame blockRBC, the compressed reconstructed frame block RBC finally being storedin the reference frame buffer SRB. To read out and process the data inthe reference frame buffer, the respective compressed reconstructedframe block RBC is fed to the frame decompression unit PD whichgenerates therefrom by decompression the reference sub-frame REFT,hereinafter also referred to as second frame data. The subsequent methodin FIG. 3 is similar to that of FIG. 1 and will not therefore bedescribed in further detail here.

FIG. 4 shows the use of the proposals on a decoder or more specificallyas part of a decoding method, wherein the setup according to FIG. 4 isidentical to the setup according to FIG. 2 apart from one modification.In contrast to FIG. 2, in FIG. 4 the compression filter CF is connectedinto the signal processing directly upstream of the frame compressionunit PC in the feedback loop. Modified first frame data RBMX is producedfrom the first frame data RBM by using the compression filter CF. Thefurther steps are similar to those described in connection with FIGS. 2and 3.

To ensure that the encoder in FIG. 3 and the decoder in FIG. 4 operatewithout drift, the respective compression filter CF shown is identicalin both versions.

The compression filter CF is configured such that, prior to compressionby the frame compression unit, it transforms the first frame data suchthat an increased compression rate can be produced in the subsequentprocessing step of the frame compression unit. In an exemplaryembodiment of the compression filter CF, the latter is designed suchthat image details can be reduced in a controlled manner by acompression filter parameter. The compression filter can be implementedin the form of a low-pass filter for which the cutoff frequency can beset using the filter parameter and which, by its filter characteristic,filters out high-frequency components in the first frame data. Filteringout the high-frequency frame components enables the frame compressionunit to compress the modified first frame data with a higher compressionrate than according to the related art. In an alternative development,the compression filter CF is designed in the form of a subband filterwhich performs the transformation of the first frame data, in particularwith suppression of quantization noise contained in the first frame dataas the result of quantization.

In addition to changing the image sharpness or rather image detailswithin the first frame data, wherein the number of pixels of the firstframe data and of the modified first frame data are identical, formatreduction can also be initiated, as will now be carried out in FIGS. 5and 6, i.e. a reduction in the number of pixels by the compressionfilter.

To this end, FIG. 5 shows, in the feedback loop, signal processing ofthe first frame data RBM by the compression filter CF, the framecompression unit PC and storage in the reference frame buffer SRB asalready explained in detail in FIG. 3. The difference with respect toFIG. 3 is now that, when the second frame data is retrieved, arespective compressed reconstructed frame block RBC is first read out ofthe reference frame buffer, and converted by the frame decompressionunit PD into modified second frame data RBCY and by inversetransformation of the modified second frame data into the referencesub-frame REFT by a decompression filter DF. For example, thecompression filter CF reduces a frame size of 100×100 pixels of thefirst frame data to a frame size of 50×50 pixels of the modified firstframe data. This pixel reduction is performed, for example, by a 2×2filter which multiples an amplitude value of each of the four pointstaken into account by a weight of 0.25 and totals the resulting weightedamplitude values. This produces from 2×2 pixels a new pixel of themodified first frame data. The decompression filter produces 100×100pixels from 50×50 pixels, e.g. by copying a pixel to each position of2×2 pixels of the reconstructed pixel. Further methods for reducing andincreasing the format will be known to an average person skilled in theart of image signal processing and will not therefore be discussed ingreater detail here.

FIG. 6 shows the variant of FIG. 5 with reference to a decoder. The modeof operation of the variant is similar to that explained in connectionwith FIG. 5.

In another embodiment of the compression and decompression filter, theamplitude values of the pixels are reduced, e.g. by quantization, orincreased, e.g. by inverse quantization.

In a particular design variant, the pair of compression anddecompression filters operate losslessly, i.e. compression filter (CF)

decompression filter (DF)=1, i.e. this linking produces a unity matrix,wherein the symbol

corresponds to a convolution or a product.

In addition to the previous examples, the compression filter andpossibly the decompression filter can be present in a plurality ofdesign variants. For example, 5 low-pass filters with different cutofffrequencies can be available for selection. Prior to using one of thedesign variants of the compression filter, it is first determined foreach variant of the compression filter the compression rate versus imagequality tradeoff thereby produced by the respective compression filterand the frame compression unit. For this purpose the associated costvalue is determined for each variant using a cost function. When thecost values for all the variants of the compression filter areavailable, the variant which achieves the lowest cost value is selected.

The respective device and method shown in the FIGS. 1-6, represented bythe respective squares, can be realized in software, hardware or acombination of software and hardware. Thus they can be stored in programcode in a program memory and read out and processed by a processorconnected to the program memory and an interface for transmitting andreceiving data such as the frame block or the output signal.

The invention has been described in detail with particular reference topreferred embodiments thereof and examples, but it will be understoodthat variations and modifications can be effected within the spirit andscope of the invention covered by the claims which may include thephrase “at least one of A, B and C” as an alternative expression thatmeans one or more of A, B and C may be used, contrary to the holding inSuperguide v. DIRECTV, 69 USPQ2d 1865 (Fed. Cir. 2004).

1-8. (canceled)
 9. A method for transforming first frame data,comprising: compressing the first frame data by a frame compressionunit; prior to compression, transforming the first frame data with acompression filter to thereby increase a compression rate of the framecompression unit; storing the first frame data in compressed from, in areference frame buffer; retrieving the first frame data in compressedform from the reference frame buffer; and decompressing the first framedata in compressed form into second frame data by a frame decompressionunit.
 10. The method as claimed in claim 9, wherein the compressionfilter transforms the first frame date as a function of at least one ofthe following parameters: a coding mode used for encoding the firstframe data; a quantization parameter used encoding the first frame data;and a motion vector used for predictive coding of the first frame data;11. The method as claimed in claim 9, wherein the compression filtercomprises a subband filter, which suppresses quantization noisecontained in the first frame data as a result of quantization.
 12. Themethod as claimed in claim 10, wherein the compression filter optimizesa rate distortion function at a predefinable data rate of the firstframe data.
 13. The method as claimed in claim 10, further comprising:comparing a plurality of filters; determining which of the plurality offilters best optimizes a rate distortion function at a predefinable datarate of the first frame data; and for the compression filter, selectingthe filter which best optimizes the rate distortion function.
 14. Themethod as claimed in claim 9, wherein decompressing the first frame datacomprises: decompressing the first frame data in compressed form in theframe decompression unit to produce decoded frame data; and subjectingthe decoded frame data to a decompression filter to generate the secondframe data, where reverse steps to the compression filter are carriedout.
 15. The method as claimed in claim 9, wherein decompressing thefirst frame data comprises: decompressing the first frame data incompressed form in the frame decompression unit to produce decoded framedata; and subjecting the decoded frame data to a decompression filter togenerate the second frame data, and the compression filter and thedecompression filter execute at least one of the following operations: afirst number of pixels of the first frame data is reduced by thecompression filter and a second number of pixels of the second framedata is increased by the decompression filter such that the first numberand the second number assume an identical value; and a third number ofbit planes of the pixels of the first frame data is reduced by thecompression filter and a fourth number of bit planes of the pixels ofthe second frame data is increased by the decompression filter such thatthe third number and the fourth number assume an identical value.
 16. Adevice to transform first frame data, comprising: a frame compressionunit to compress the first frame date; a compression filter to transformthe first frame data prior to compression, to thereby increase acompression rate of the frame compression unit; a reference frame bufferto store the first frame data in compressed form; and a framedecompression unit to decompress the first frame data in compressedform, after retrieval of the first frame data in compressed form fromthe reference frame buffer, the first frame data in compressed formbeing decompressed into second frame data.
 17. The device as claimed inclaim 16, wherein the frame decompression unit decompresses the firstframe data in compressed form to produce decoded frame data, the devicefurther comprises a decompression filter to subject the decoded framedata to decompression filtering to generate the second frame data, andthe decompression filter operates inversely to the compression filter.